State Based Robustness Testing for Components

Component based development allows to build software upon existing components and promises to improve software reuse and reduce costs. To gain reliability of a component based system, verification technologies such as testing can be applied to check underlying components and their composition. Conformance testing checks the consistency between the behavior and component specifications. On the other hand, robustness testing detects vulnerability of software with unexpected input or stressful environment. Existing robustness testing tools aim to crash components with preset values of different data types. But they do not take into account component states, which are vital to the detecting robustness problem of a component. We propose a state machine based approach to detect robustness problems of components. Firstly, a set of paths is generated to cover transitions of the state machine. Test inputs which follow the paths achieve high coverage of the system states and examine more transitions than stateless API testing. Secondly, invalid inputs and inopportune method calls are fed to the component in different states to test the robustness. When unexpected exceptions arise in the test runs, robustness failures are reported. We do a case study on a component from an open source software and it results in positive results.